Universal Finite Memory Coding of Binary Sequences

This work considers the problem of universal coding of binary sequences, where the universal encoder has limited memory. Universal coding refers to a situation where a single, universal, encoder can achieve the optimal performance for a large class of models or data sequences, without knowing the model in advance, and without tuning the encoder to the data. In the previous work on universal coding, specific universal machines, whose performance attained the theoretical limits, were suggested. However, these machines require unlimited amount of memory. This work investigates the case where the universal machines are with limited resources, i.e., finite-state machines. To simplify the problem, this work considers universal finite-state machines that assign probabilities to the next bit. It ignores the additional number of states needed to translate the assigned probabilities into code bits. In most cases examined, this work provides lower bounds on the performance, which describe the optimal rate (as a function of number of states) that the entropy, or the empirical entropy, can be attained. In addition, in most cases, this work presents specific machines whose performance is compared to these bounds. While the general problem of universal coding with limited resources is still open, this work provides a set of basic results that will be useful in analyzing the general problem. These results thus provide an important step in understanding the problem of limited resources universal coding.